Skip to main content

Download notebook · View source · Open in Colab

Working with Bounding Boxes and Polygons

Bounding boxes and polygons attach labeled regions to images. Queries can select regions by their properties, follow their connections to source images, and compare their geometry. The image ROI reference and video ROI reference describe the supported commands.

Required demo environment​

These examples read a prepared COCO database and require the demo notebook environment's helper module. Configure the connection with APERTUREDB_JSON, APERTUREDB_KEY, or an active SDK profile. The queries require object read permission and do not create or delete data.

The database must contain COCO images with adb_data_source: "coco" and adb_image_id, bounding-box annotations, and polygons with ann_id. The examples select image 458992, polygon 226510, and labels horse, dog, tie, and person. These selectors must match annotations present in the database.

The COCO deployment sources describe the prepared dataset. Their data scripts read the ad-demos-datasets Google Cloud Storage bucket or the aperturedb-demos S3 bucket. Access to that data and the deployment-specific helper is required; installing the SDK alone does not provide them.

The bounding boxes and polygons example supplies a public image and annotations for a separate local example.

# The prepared demo environment must provide this notebook's helper module.
from importlib.util import find_spec
if find_spec("helper") is None:
    raise RuntimeError(
        "PolygonExamples requires the prepared COCO demo environment with its helper module. "
        "See Required demo environment above; use the Regions notebook for a standalone local example."
    )

# Dependencies
from aperturedb.Images import Images
from aperturedb.Utils import Utils
from aperturedb.CommonLibrary import create_connector
from aperturedb import NotebookHelpers as nh
from aperturedb.Constraints import Constraints
import os

import helper

# Connect to the ApertureDB instance.
db = create_connector()
utils = Utils(db)
status, _ = db.query([{"GetStatus": {}}])
assert isinstance(status, list) and status[0]["GetStatus"]["status"] == 0, status


# Encode literal data without interpreting a leading dollar sign as a property.
def literal(value):
    if isinstance(value, str) and value.startswith("$"):
        return "$" + value
    if isinstance(value, (list, tuple)):
        return {"array": [literal(item) for item in value]}
    if isinstance(value, dict):
        if len(value) == 1 and ("date" in value or "_date" in value):
            return {"to_date": literal(next(iter(value.values())))}
        return {"json": value}
    return value
# Parameters used across queries in this notebook.
demo_name = "coco"
data_source   = "coco"
width         = 600
height        = 600
display_limit = 1
label         = "horse"
polygon_label = "dog"
roi_label1    = "tie"
roi_label2    = "person"
image_id      = 458992
polygon_id    = 226510

Display BoundingBoxes using the Python object mapper API​

The Python object wrapper retrieves images and displays their bounding-box annotations.

print("\nShowing bounding boxes overlaid on a subset of images from the dataset: " + data_source)

imgs  = Images(db, batch_size=display_limit)
const = Constraints()
const.equal("adb_data_source", data_source)

imgs.search(constraints=const)
print("where " + str(imgs.total_results()) + " images matched the query\n")

# It is as simple
# Note: The images are not ordered, and may not have associates bounding boxes.
# those will be diplayed, but not highlighted.
imgs.display(limit=display_limit, show_bboxes=True)

Display Polygons​

The wrapper can also overlay polygon annotations on the source images.

# Simply switch over to a polygon display
imgs.display(limit=display_limit, show_polygons=True)

Find Images using Labels or Annotations​

The following JSON queries combine annotation filters, graph traversal, and image retrieval in a single request.

Retrieve images with a labeled region inside a rectangle​

Graph traversal connects region matches to their source images. This query selects labeled bounding boxes inside a rectangle, finds connected images with the requested dimensions, and resizes the returned images.

print("\nReturning resized version of images with resolution better than: " + str(width) + "x" + str(height) + " with label: " + label + "\n")

q = [ {
    "FindBoundingBox": {
        # this is key to traversing the graph-based metadata. We are declaring a reference.
        "_ref": 1,
        "with_label": label,
        # ensuring that the box with label is in a given area of the image
        "in_rectangle": {
            "x": 0,
            "y": 100,
            "width": 300,
            "height": 500
        },
        # we would like to see the coordinates of this box
        "coordinates": True,
        # but not retrieve the pixel area contained within
        "blobs": False,
        "labels": True,
    }
}, {
    "FindImage": {
        "is_connected_to": {"ref": 1},
        "uniqueids": True,
        "constraints": ['all', ['$width', '>', literal(width)], ['$height', '>', literal(height)]],
        "operations": [{"type": "resize", "width": 224, "height": 224}],
        "blobs": True,
        "limit": display_limit
    }
} ]

responses, blobs = db.query(q)

#Let's make a wrapper with the response from db.query
assert isinstance(responses, list) and responses[1]["FindImage"]["status"] == 0, responses
images_wrapper = Images(db, batch_size=display_limit,
                        response=responses[1]["FindImage"].get("entities", []),
                        blobs=blobs, query=q[1]["FindImage"])

# and display the results with bounding boxes drawn on top
images_wrapper.display(show_bboxes=True)
# Show the above again, with polygons instead of bounding boxes
images_wrapper.display(show_polygons=True)

Compare regions with intersection over union​

Filter and sort regions by their properties​

FindPolygon filters polygon annotations by their properties. This example selects polygons with the requested label on high-resolution images and displays them in descending order of area, measured in pixels.

images, large_dog_polygons = helper.find_polygons_by_property(db, with_label=polygon_label, property="_area", sort="descending", limit=display_limit)

print("displaying the {} largest {} polygons".format(display_limit, polygon_label))

images.display(polygon_constraints=large_dog_polygons, polygon_tag_key="_area", polygon_tag_format="{:.0f} pixels")

Find overlapping polygons​

ApertureDB can select polygon regions that intersect other regions in the same image. This query finds polygons labeled roi_label1 that overlap a polygon labeled roi_label2.

print("\nReturning images with annotated polygon: " + roi_label1 + " overlapping with polygon: " + roi_label2 + "\n")

images, overlapping_polygons = helper.find_overlapping_polygons(db, roi_label1, roi_label2)

images.display(polygon_constraints=overlapping_polygons, limit=display_limit)

Find polygons in a fixed location​

Polygon queries can compare annotations with a fixed region of an image. This supports tasks such as selecting people near a doorway or objects extending beyond the frame.

The example defines an r-shaped region around the image's upper-left corner, shown in red, and finds polygons labeled polygon_label that intersect it.

upper_left_corner_region = [ # This describes the following shape:
    [0.0, 0.0],              #    ____________
    [100.0, 0.0],            #   |    _______/
    [95.0, 5.0],             #   |  /
    [40.0, 5.0],             #   | |
    [5.0, 40.0],             #   | |
    [5.0, 95.0],             #   | |
    [0.0, 100.0],            #   |/
]

print("\nReturning images where annotated polygon: " + polygon_label + " spans the upper left corner of the image\n")

images, trains_in_upper_left = helper.find_polygons_within_region(db, with_label=polygon_label, within_region=upper_left_corner_region)

images.add_polygon_overlay([upper_left_corner_region],(0,0,255))

images.display(polygon_constraints=trains_in_upper_left, limit=display_limit)

Evaluating an Object Detector by its RoI quality​

RegionIoU compares region geometry through intersection over union. This example computes pairwise IoU values between bounding boxes and polygons in the same image to measure how closely their regions agree.

print("\nIoU matrix for an image with id : " + str(image_id) + "\n")

image = helper.print_iou_matrix_for_image(db, image_id)

image.display(show_polygons=True)
image.display(show_bboxes=True)

Polygon Levels of Detail​

FindPolygon can return several representations of a polygon's geometry:

  • bounds: the smallest axis-aligned bounding box containing the region.
  • hulls: the smallest convex polygon containing the region.
  • vertices: the full polygon geometry as it was added.
  • decompositions: the region split into disjoint convex pieces.

The output shows each representation for one polygon.

print("\nPolygon geometry for a specific polygon with id : " + str(polygon_id) + "\n")

helper.display_polygon_levels_of_detail(db, polygon_id, "ann_id")